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ABSTRACT 



In September 1993 the U.S. Department of Education (ED) 
released a handbook to assist school- and community-based practitioners in 
designing and conducting evaluations of drug- and violence-prevention 
programs funded under the Drug Free Schools and Communities Act (DFSCA) . A 
study was undertaken to assess the level of customer satisfaction with the 
handbook, and to explore the broader context in which local evaluations of 
alcohol -and-other- drug use and violence-prevention programs occur. Data were 
obtained through telephone interviews with DFSCA officials from a nationally 
representative sample of 550 school districts. The survey elicited 506 
responses, a 95 percent response rate. The weighted data represent the nearly 
14,000 local educational agencies (LEAs) throughout the United States. In 
sum, most local program coordinators who were responsible for evaluation of 
DFSCA programs acknowledged the importance of evaluation to effective 
operations. However, when faced with limited time and funds, many 
coordinators reported that they would rather allocate their resources to 
student services than to formal program evaluation. In general, program 
coordinators perceived guidance and direction from state education agencies, 
other technical-assistance providers, and written materials to be helpful. 
However, state and federal expectations for local evaluation need to be more 
clearly and consistently defined. It is recommended that ED work to establish 
uniform data- collection requirements for local education agencies that are 
clearly articulated, easy to understand, and implemented through a targeted 
technical-assistance initiative. Eight tables are included. The appendix 
contains tabulated survey responses. (LMI) 
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Executive Summary 



In September 1993 the U.S. Department of Education (ED) released a handbook to assist 
school- and community-based practitioners in designing and conducting evaluations of drug and 
violence prevention programs fimded under the Drug-Free Schools and Commimities Act 
(DFSCA). The handbook, entitled Understanding Evaluation: The Way to Better Prevention 
Programs, is designed to be an aid in improving the quality of local evaluations and thereby 
enhance state and local capacity to operate effective programs. ED contracted with the Research 
Triangle Institute (RTI) in October 1994 to assess the level of customer satisfaction with the 
handbook, and to explore the broader context in which local evaluations of alcohol and other 
drug (AOD) use and violence prevention programs occur. The specific purposes of RTFs task 
order were to: 

■ Identify the types of data collection activities that local educational agencies (LEAs) 
typically conduct; 

■ Describe the sources and types of technical assistance LEAs need and obtain to help 
them conduct high-quality evaluations; 

■ Determine the level of LEA familiarity and satisfaction with the evaluation handbook 
and other resources; and 

■ Obtain the perceptions of program staff at the local level regarding impediments to 
more successful evaluations, and recommended improvements. 

We obtained the information needed to address study purposes through telephone 
interviews with DFSCA officials from a nationally representative sample of 550 school districts, 
and achieved a 95 percent response rate to the survey, with a total of 506 completed interviews. 
We weighted the data to represent the nearly 14,000 LEAs nationwide. We briefly summarize 
the findings of this investigation below. 



Local Data Collection Activity 

■ The frequency with which LEAs conduct various forms of data collection declines as 
the complexity of the evaluation design increases: 93 percent of LEAs described 
program activities and assessed the quality of implementation, while just 10 percent 
of LEAs implemented an experimental design to compare outcome measures for a 
treatment and a control group. 

■ Eighty-six percent of LEAs conducted surveys of student AOD use during the last 
two years, 48 percent in response to state mandates. 
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Executive Summary 



■ Other types of data frequently collected by LEAs are those needed to describe 

program operations and assess the scope of populations at risk of AOD use or violent 
behavior. 



Local Uses of and Needs for Technical Assistance 

■ State educational agencies are the most frequent providers of technical assistance to 
LEAs on evaluation issues, with half of all districts reporting receipt of assistance 
from their state educational agency (SEA); the five DFSCA Regional Centers' 
provided technical assistance on evaluation issues to approximately one-fourth of all 
LEAs. 

■ The most frequent type of assistance provided by all sources of technical assistance 
consisted of workshops or seminars on program issues in general that did not focus on 
any one dimension of program evaluation; SEAs were more likely to provide 
assistance in the preparation of grant applications, the Regional Centers in the 
identification of curricular and other materials, independent professionals and 
universities in the actual conduct of evaluations, and other state and local agencies in 
issues related to the DARE program. 

■ Evaluation planning, obtaining or developing data collection instruments, and data 
analysis are the aspects of program evaluation in which most LEAs have the greatest 
need for assistance. 



LEA Familiarity and Satisfaction With the Evaluation Handbook 

■ One-third (33 percent) of all local DFSCA program coordinators in the nation are 
familiar with the evaluation handbook, and 21 percent have actually read the 
document. 

■ Features of the handbook most often identified by local coordinators as especially 
useful include (1) a good overview of evaluation, (2) ease of understanding, and (3) 
its practical orientation. 

■ Of those coordinators who read the handbook, 21 percent found it to be "very useful" 
in evaluating their DFSCA program and another 76 percent found it to be "somewhat 
useful." Information from the handbook has been helpful across all areas of program 
evaluation, especially in evaluation planning and design. 



^The 1994 reauthorization consolidated the DFSCA Regional Centers with other categorical technical 
assistance centers into a new program of Comprehensive Regional Assistance Centers. The new Comprehensive 
Centers became operational on April 1, 1996. 
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Executive Summary 



Impediments to More Successful Evaluation and Recommendations for Improvement 

■ When asked directly about their views on the importance of evaluation, three-fourths 
of local coordinators agree that formal evaluation is essential to effective program 
operations; however, when asked the same question in the context of limited 
resources, the relative priority of evaluation is diminished in the views of many local 
coordinators. 

■ Although a majority of program coordinators believe it is reasonable to expect local 
staff to have the expertise needed to conduct high-quality evaluations of their DFSCA 
programs, 47 percent cited a lack of expertise in evaluation as a significant 
impediment to the more successful evaluations. 

■ Local program coordinators identified lack of time, lack of money, lack of expertise 
in evaluation, and unclear expectations from state and federal governments as the 
greatest impediments to more successful local evaluation. 

■ In response to an open-ended question, local coordinators recommended 
improvements to local evaluations. The most frequent recommendations offered by 
local coordinators who identified specific improvements include (1) more training for 
district staff (25 percent), (2) better guidance from state and federal governments on 
expectations for DFSCA program evaluation (21 percent), (3) more funding (18 
percent), (4) more technical assistance (15 percent), and (5) establishment of uniform 
data collection requirements for LEAs (14 percent). 

In sum, most local program coordinators who are responsible for evaluation of DFSCA 
programs acknowledge the importance of evaluation to effective operations. For example, nearly 
all coordinators reported that evaluation data were useful in identifying student needs, directing 
changes in program content and delivery, justifying continued funding, and fulfilling state and 
local requirements.^ Particularly when faced with limited time and funds, however, many 
coordinators reported that they would rather allocate their resources to student services than to 
formal program evaluation. 



In general, the guidance and direction provided by SEAs, other technical assistance 
providers, and written materials such as the evaluation handbook are perceived to be helpful, but 



^Other recent studies have found that only about half of selected districts reported conducting formal 
evaluations of their prevention programs, and even fewer reported use of evaluation findings for program 
improvement purposes (see Silvia, E.S., and Thome, J.M. School-Based Drug Prevention Programs: A 
Longitudinal Study in Selected School Districts. Prepared under contract to the U.S. Department of Education. 
Research Triangle Park, NC: Research Triangle Institute, 1997). 

O 
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not as helpful as they might be if state and federal expectations for local evaluation were made 
clearer and remained consistent from year to year. The problems of limited resources and 
confusion over expectations would be ameliorated by the establishment of uniform data 
collection requirements for LEAs that were clearly articulated, easy to understand, and 
implemented through a targeted technical assistance initiative. ED's ongoing work in the 
development of program performance indicators and a model data collection system for 
obtaining the data needed to implement the indicators would appear to be a major step toward 
creating the conditions under which the quality of local Safe and Drug-Free School program 
evaluations will improve. 
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LOCAL EVALUATION OF PROGRAMS FUNDED UNDER THE 
DRUG-FREE SCHOOLS AND COMMUNITIES ACT 



Under the Drug-Free Schools and Communities Act or DFSCA (which was reauthorized 
in 1994 as the Safe and Drug-Free Schools and Communities Act, or SDFSCA), the U.S. 
Department of Education (ED) annually distributes funding to state education agencies (SEAs) to 
support the establishment and operation of drug prevention programs.' SEAs, in turn, award 
grants to local educational agencies (LEAs), nearly all of which now operate prevention 
programs. While the law allows considerable flexibility in the types of activities that may be 
supported, the most common activities carried out under the DFSCA included student assistance 
programs, student instruction and training, student support groups and counseling, peer 
leadership activities, parent education, and teacher and staff training. 

In September 1993, ED released a handbook to assist school- and community-based 
practitioners in designing and conducting evaluations of prevention programs funded under the 
DFSCA. ED distributed the handbook, entitled Understanding Evaluation: The Way to Better 
Prevention Programs (Lana Muraskin, 1993), to every school district in the country in an effort 
to impress upon local practitioners the importance of conducting evaluations of their prevention 
programs.^ The handbook is designed to be an aid in improving the quality of local evaluations 
and thereby enhance state and local capacity to operate effective programs. 

ED contracted with the Research Triangle Institute (RTI) in October 1 994 to assess the 
level of customer satisfaction with the handbook. Through structured telephone interviews with 
local DFSCA program staff, RTI investigated the extent to which local DFSCA program staff are 
using the handbook, and with what effects. The interview eilso gathered information on the 
context in which local evaluations of alcohol and other drug (AOD) use prevention programs 

^SDFSCA statutory changes, including the addition of violence prevention as an allowable program 
activity, were authorized beginning in fiscal year 1 995 . 

^Although the handbook is intended as an aid to local DFSCA program evaluation, a disclaimer on the 
inside cover of the handbook notes that the views expressed "do not necessarily reflect the policy" of the 
Department. Thus, guidance provided in the handbook may not be cited as a federal mandate or prescription for 
local program evaluation. 
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occur, including local uses of, and needs for, technical assistance in evaluation. The specific 
purposes of RTl's task order were to; 

■ Identify the types of data collection activities that LEAs typically conduct; 

■ Describe the sources and types of technical assistance LEAs need and obtain to help 
them conduct successful evaluation; 

■ Determine the level of LEA familiarity and satisfaction with the evaluation handbook 
and other resources; and 

■ Obtain the perceptions of program staff at the local level regarding impediments to 
more successful evaluations, and recommendations for state and federal officials for 
program improvement. 

We obtained the information needed to address study purposes through telephone 
interviews with DFSCA officials from a nationally representative sample of 550 school districts. 
We achieved a 95 percent response rate to the survey, with a total of 506 completed interviews.^ 
The intended respondent to the interview was the LEA's DFSCA program coordinator, but while 
all respondents were responsible for administration of the DFSCA program in their LEA, most 
also functioned in other capacities, from district superintendent to school psychologist, since 
DFSCA program coordination at the local level is a full-time job in only the largest of school 
districts. A copy of the survey instrument, which includes tabulations of responses to each 
survey item, is appended to this report. In the remainder of this report we present the study's 
findings, beginning with a summeiry of local data collection activities. 

Local DFSCA Program Data Collection Activities 

The DFSCA required each SEA to submit a biennial report of its performance 
that includes an evaluation of the effectiveness of state and local programs. The law also 
required each LEA to submit an annual progress report to the SEA that includes significant 
accomplishments during the preceding year, the extent to which objectives are being achieved, 
methods used by the LEA to evaluate program effectiveness, and evaluation results. To date ED 




’We weighted the data from these 506 completed interviews to represent the 13,762 LEAs in the nation. 

V 0 



has completed three biennial surveys of state performance, covering the periods 1987-89, 1989- 
91, and 1991-93/ 

Broadly speaking, data from these surveys consistently indicate that SEAs, and LEAs, 
about whose evaluation activities SEAs report, are much more likely to conduct process 
evaluations than they are to implement formal outcome evaluations; and findings from this study 
confirm SEA reports of local practice. We asked respondents to identify the types of evaluation 
activities they had completed over the last two years (school years 1993-94 and 1994-95), and to 
indicate for each whether the district conducted the activity in response to a state mandate, or to 
fulfill local needs only. Table 1 summarizes these data. 



Table 1; Estimated Percentage of LEAs Conducting Specific Evaluation Activities^ 



Evaluation Activity 


Total 


State 

Mandated 


Local Use 
Only 


Description of program activities (n= 12,749) 


93% 


76% 


17% 


Assessment of the quality of program 
implementation (n=l 3,685) 


93% 


66% 


27% 


Comparison of pre and post assessments on the 
group receiving services (n=13,594) 


65% 


31% 


34% 


Comparison of outcome measures for local 
program participants with national or state 
averages (n=l 3,680) 


54% 


27% 


27% 


Longitudinal data collection of outcome measures 
(n= 13,676) 


53% 


31% 


22% 


Cross sectional data collection of outcome 
measures (n= 13,6 16) 


50% 


27% 


23% 


Comparison of outcome measures for a treatment 
group and a control group (n=l 3,688) 


10% 


4% 


6% 



As shown, the frequency with which LEAs conduct various forms of data collection 
declines as the complexity of the evaluation design increases. While over 90 percent of LEAs 



fourth such state survey, covering the period 1993-95, is in process. Under SDFSCA, states are required 
to report triennially. 

^Numbers and percentages of districts in all tables are estimated, based on weighted data, reflecting the 
universe of districts in the nation. 



assessed the quality of program implementation and obtained descriptive statistics on program 
activities, just 10 percent of all LEAs evaluated their programs using an experimental design. At 
least half of all LEAs conducted evaluation activities that lie somewhere between simple 
program description and experimental designs with respect to technical complexity . 

The specific information needed by an LEA to gauge the success of its AOD use program 
will vary according to the program's particular purposes and activities; however, the principal 
measure of success at local, state, and federal levels is the extent to which AOD use has declined 
over time. We know from the most recent biennial state survey that all states have conducted 
surveys of student use of alcohol and other drugs at some point since program inception, and that 
all LEAs in at least 15 states conducted prevalence surveys in 1991-93. This study of local 
evaluation practice found that over the last two years 86 percent of LEAs conducted surveys of 
student use of alcohol and other drugs, 48 percent in response to a state mandate and 38 percent 
for local use. 

Other types of data frequently collected by LEAs are those needed to describe program 
operations and to assess the scope of the populations at risk of AOD use. In the last two years at 
least 80 percent of LEAs in the nation obtained measures of truancy, dropping out of school, 
juvenile arrests, referral for AOD treatment, disciplinary actions resulting from AOD use, 
suspension or expulsion from school, and participation in AOD prevention activities. In most 
instances, SEAs require that LEAs collect these data, although the information is also clearly 
critical to local program planning and operations. Information on youth suicides or suicide 
attempts, and on source of referral, although somewhat less likely to be required by the state, are 
still collected by a majority of LEAs; measures of the extent of illegal gang activity are obtained 
by just 41 percent of school districts nationwide. 

As suggested above, nearly all LEAs (97 percent) relied on these descriptive data to 
identify student needs for AOD use and violence prevention interventions. Most school districts 
also used these data to identify necessary changes in program content and delivery (90 percent), 
and to justify continued funding (84 percent). Overall, 93 percent of LEAs used evaluation data 
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to fulfill state requirements and 82 percent to respond to local requirements. That such a high 
percentage of districts reported using these data to fulfill local needs, as well as to respond to 
state mandates, suggests that in most localities evaluation activity has an immediate and directly 
relevant purpose. 

Local Views on the Role of Evaluation and Needs for and Use of Technical Assistance 

To assess the extent to which local DFSCA coordinators do, in fact, perceive evaluation 
as useful for their own immediate purposes, and thus have a vested interest in the quality of 
evaluation that extends beyond compliance with state and federal regulations, we asked 
respondents to indicate their level of agreement with several statements concerning the 
importance of evaluation. Overall, three-quarters (74 percent) of DFSCA coordinators agreed 
that "formal evaluation is an essential component of a successful drug and alcohol prevention 
program." Moreover, when asked to respond to the statement "evaluation is necessary primarily 
to document what we are doing for the state or federal government, rather than for meeting local 
needs," only 26 percent agreed, while 64 percent disagreed and 10 percent were neutral. 

Not only do a majority of program officials apparently believe evaluation to be critical to 
successful AOD and violence prevention programs, but most also believe that they should be 
responsible for conducting the evaluations. Anecdotal evidence from prior studies of the DFSCA 
program has suggested that many local officials feel ill-equipped to implement formal evaluation 
designs and that many would prefer to have an independent agency or professional evaluate their 
programs. However, when asked about these issues directly, a majority of DFSCA coordinators 
believe it is reasonable for the state and federal government to expect them to have the expertise 
needed to conduct quality evaluations (59 percent) and that program staff, rather than an 
independent entity, should conduct the evaluation (72 percent). 

At the same time, local officials acknowledge their need for assisteince in virtually all 
aspects of program evaluation. Over half of all program coordinators indicated a need in their 
district for technical assistance in the areas of evaluation planning (61 percent), obtaining or 



developing data collection instruments (60 percent), data analysis (59 percent), and using 
evaluation findings for program improvement (54 percent). A significant percentage of program 
officials also perceive a need for assistance with reporting of evaluation findings (43 percent) and 
data collection (40 percent). Asked to identify the area in which they most needed assistance, 
nearly one-third indicated "obtaining or developing data collection instruments," 24 percent 
identified "evaluation planning," and 22 percent reported "using evaluation results to improve the 
program." 

To identify the extent to which LEAs avail themselves of existing technical assistance 
resources, we asked respondents to indicate if they had received any type of assistance in 
evaluation issues during the past year (school year 1 994-95) from a variety of sources. Table 2 
indicates the percentage of all LEAs that obtained training or technical assistance from specific 
sources during the past year, and the percentage who identified each source as the most useful. 



Table 2: Estimated Percentage of LEAs Obtaining Training and Technical Assistance 

from Specific Sources During the Past Year 



Source of Assistance 


Percentage of LEAs 
Reporting Receipt of 
Technical Assistance 


Percentage of LEAs 
Identifying Each 
Source as the Most 
Useful Assistance 


State educational agencies (n= 13,707) 


50% 


39% 


DFSCA Regional Centers (n= 13,648) 


26% 


12% 


Independent professional (n=l 3,485) 


24% 


18% 


Other state agency (n= 13,666) 


20% 


9% 


College or university (n=13,719) 


14% 


4% 


Other source* (n=l 1,369) 


24% 


17% 



As would be expected, SEAs were the most frequent source of technical assistance on 
evaluation issues at the local level, with half of all respondents reporting receipt of technical 



®Other sources included law enforcement agencies, local social service agencies, and private treatment 

facilities. 
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assistance from their SEA during the last school year. The last biennial survey of SEAs revealed 
that in school year 1991-92 SEAs allocated approximately $5.6 million of their 10 percent set- 
aside funds (28 percent of the total $19.8 million set aside) to provide training and technical 
assistance to school and community -based AOD program staff. How much of this assistance 
targeted evaluation issues specifically is not known; however, 91 percent of SEAs provided 
technical assistance to LEAs on evaluation issues.’ 

The second most frequent source of technical assistance in evaluation issues was the five 
DFSCA Regional Centers, from whom approximately one-fourth of all LEAS received training 
or technical assistance.* The DFSCA Regional Centers assisted schools and communities to 
operate effective AOD prevention programs through training, technical assistance, dissemination, 
and in responding to SEA and LEA requests. Independent professionals or consultants were 
used by nearly as many LEAs to obtain technical assistance in evaluation issues, as were "other" 
sources, the bulk of which are local law enforcement and other community-based agencies. One- 
fifth of all LEAs obtained assistance from a state agency other than the SEA, and 14 percent 
from a college or university. 

To some extent the most appropriate provider of technical assistance will depend on the 
type of help an LEA needs, although the most frequent type of assistance provided by all sources, 
as reported by local progrtun coordinators, consisted of workshops or seminars on program issues 
in general that did not necessarily focus on any one dimension of program evaluation. F or SEAs 
the second most frequent type of assistance provided to LEAs was help in the preparation of 
grant applications, while the second most frequent type of assistance provided by Regional 
Centers was in helping LEAs to identify appropriate curricular and other materials. Independent 
professionals and colleges or universities were more likely to assist in the actual conduct of 



’Characteristics of DFSCA State and Local Programs, 1991-93. 

®The 1994 reauthorization consolidated the DFSCA Regional Centers with other categorical technical 
assistance centers into a new program of Comprehensive Regional Assistance Centers. The new Comprehensive 
Centers became operational on April 1, 1996. 



program evaluation than were other sources, while other state and local agencies were more 
likely to provide assistance related to the Drug Abuse Resistance Education (DARE) program. 

LEAs' Use of The Evaluation Handbook and Other Resources 

The evaluation handbook ED disseminated to all LEAs in 1993 is designed as a resource 
document for local staff to consult on many of the same issues addressed by SEAs, the Regional 
Centers, and other technical assistance providers in their efforts to assist LEAs to conduct more 
rigorous evaluations. The handbook stresses the importance of evaluation to effective AOD 
prevention programs, introduces the reader to a few basic principles of successful evaluation, and 
describes the steps involved in designing and conducting program evaluations using case 
examples drawn from a fictitious school district. The handbook's imderlying premise is that 
"many evaluations that use simple designs can be conducted without formal training in 
evaluation," and it stresses the integration of evaluation into the overall fabric of a program and 
the involvement of program personnel (as opposed to independent experts) in decision making 
regarding the evaluation's goals, conduct of the evaluation, and interpretation of evaluation 
results. 



One-third (33 percent) of all local DFSCA coordinators in the nation were familiar with 
the handbook, and 21 percent have actually read the document.^ We asked respondents who 
have read the handbook to identify those features of the document that they particularly liked, 
and Table 3 summarizes the responses of those who were able to identify specific features. Of 
the approximately 3,000 coordinators who read the handbook, 64 percent (or 13 percent of all 
LEAs) reported that they have applied information from the handbook in their work. 

Table 4 summarizes the ways in which local program coordinators reported using 
information from the handbook. As these data show, information from the handbook has been 



®Given the time lag between ED's dissemination of the document and the initiation of our interviews 
approximately two years later, it is reasonable to assume that these percentages somewhat underestimate actual 
awareness of the document, as a result of staff turnover and other intervening factors. 

‘“Twenty-seven percent of the respondents who had read the handbook were unable to remember any 
particular feature. 
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helpful across all areas of evaluation, especially in the planning and evaluation design phases. 
Overall, of those who used information from the handbook, 21 percent reported that it was "very 

Table 3: Estimated Percentage of Local Coordinators Reporting Specific Features 

of the Handbook as Especially Useful 



Handbook Features 


Of Coordinators Who Read the 
Handbook, Percentage Who 
Cited Each Factor (n=2,934) 


Provides a good overview of evaluation 


29% 


Easy to understand 


26% 


Practical orientation 


25% 


Use of case examples 


13% 


Suitability as a reference document 


9% 



Table 4: Estimated Percentage of Local Coordinators Who Read the Handbook Who 
Applied the Information in Various Aspects of Local SDFSCA Program Evaluation 



Uses 


Of Coordinators Who Read the 
Handbook, Percentage Who Applied the 
Information (n= 2,934) 


Evaluation planning or design 


88% 


Identifying evaluation questions or issues 


79% 


Interpreting data collected 


62% 


Reporting of evaluation findings 


59% 


Implementation of the evaluation design 


58% 


Staff training 


47% 


Developing data collection instruments 


45% 



useful" in evaluating their DFSCA programs, and another three-fourths of these respondents (76 
percent) reported the handbook as being "somewhat useful" in evaluation efforts. Only two 
percent of those who read the document believed the handbook to be "not useful." 



An appendix to the handbook identifies other guides to AOD use prevention program 
evaluation potentially useful to local staff in the evaluation of their DFSCA programs, and we 



asked respondents to indicate their familiarity with each. Table 5 identifies the percentage of 
local program coordinators familiar with each document. Useful features of these resources, as 
identified by respondents, include many of the same characteristics cited about the ED handbook, 
such as ease of understanding, use of examples, and suitability as a reference document. 



Table 5: Estimated Percentage of Local Coordinators Familiar 
With Other Evaluation Guides 



Resource 


Percentage of 
Coordinators 
(n=13,748) 


Prevention Plus III, Assessing Alcohol and Other Drug Prevention 
Programs at the School and Community Level (Linney and Wandersman) 


20% 


Handbook for Evaluating Drug and Alcohol Prevention Programs 
(STEPP) (University of Maryland, 1990) 


16% 


Evaluation of Prevention Programs: A Basic Guide for Practitioners 
(Moberg, 1986) 


12% 


CSAP Technical report #8: Measurements and Prevention 


7% 


Program Evaluation Handbook: Drug Abuse Education (lOX Associates) 


7% 



Impediments to Local Evaluation and Recommendations for Improvement 

We asked respondents to identify significant impediments to successful evaluation of 
their districts’ DFSCA program, and not surprisingly, the two most frequently cited were a lack 
of time (81 percent) and a lack of money (69 percent). As indicated in Table 6, a lack of 
expertise needed to conduct successful evaluations and unclear expectations from state and 
federal governments were also identified as significant impediments by nearly half of all 
respondents, followed by confidentiality issues, and lack of support from district officials. 
Asked to identify the greatest single impediment, 37 percent cited a lack of time, another third 
(34 percent) reported a lack of money, and 1 0 percent unclear expectations from the federal 
government. 
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That "a lack of time" is identified as the greatest impediment to successful evaluation by 
more respondents that any other single factor reflects the fact that most local program 



Table 6: Estimated Percentage of Coordinators Citing Specific 
Impediments to More Successful Local DFSCA Program Evaluation 



Impediment 


Percentage of 
Local Coordinators 


Lack of time (n=13,762) 


81% 


Lack of money (n=13,755) 


69% 


Lack of expertise in evaluation (n= 13,740) 


47% 


Unclear expectations from state/federal government (n= 13,725) 


45% 


Lack of support from district officials/low priority (n=l 3,730) 


24% 


Confidentiality issues (n=l 3,762) 


23% 



coordinators allocate only a portion of their time to DFSCA-related activities in general, let alone 
DFSCA program evaluation. Previous reports of the DFSCA program" have noted that 
responsibility for program administration and operation at the local level is assumed by a wide 
range of local officials, from district superintendent to school nurse, and the respondents to our 
telephone interview also varied considerably with respect to job title, as indicated in Table 7. 



Table 7: Estimated Percentage of Respondents With Specific Job Titles 



Job Title 


Percentage of 
Respondents (n=13,762) 


Prevention program coordinator 


24% 


District superintendent or assistant superintendent 


19% 


School counselor, psychologist, or social worker 


16% 


School principal or assistant principal 


13% 


Other local administrator 


10% 


Instructor 


7% 


Director of student services 


5% 


Student assistance program coordinator 


3% 


Other 


3% 



"LEA Cross-Site Analysis. 



As may be discerned from the variety of job titles listed, program administration and evaluation 
constitute a small percentage of overall responsibilities for many local program coordinators. 
Moreover, these individuals are essentially concerned with service delivery and one would not 
expect service delivery staff to place as high a priority on evaluation as would professional 
researchers or evaluation specialists, particularly given limited resources. 

When asked directly if formal evaluation is essential to a successful program, nearly 
three-fourths of respondents agreed that it is, as we have noted; however, when the same issue is 
raised within the context of limited resources and time, the relative priority of formal evaluation 
declines in the views of a sizeable percentage of local coordinators. For example, a majority of 
respondents (53 percent) agreed with the statement "our district would rather allocate all of its 
DFSCA resources to program development and student services than allocate up to five percent 
to evaluation." While it is understandable that local administrators place a high priority on 
service delivery, these data suggest that many local coordinators may not fully appreciate the 
value of formal evaluation to effective program operations. Further evidence of this notion is 
reflected by the 44 percent of respondents who agreed with the statement, "my district is able to 
meet its evaluation needs through ongoing observation of program operations and activities; 
formal evaluation is not necessary.'"^ 

Finally, even though most respondents believe it is reasonable to expect DFSCA program 
staff to have the expertise needed for planning and conducting formal evaluation, as we noted 
earlier, many clearly do not believe that they possess the requisite skills. In fact, 47 percent of 
respondents cited a lack of expertise in evaluation as an impediment to successful evaluation, and 
the percentages of respondents reporting a need for technical assistance in six discrete areas of 
evaluation (e.g., planning, collecting data, analysis, etc.) ranged from 40 to 61 percent. 

As Table 6 indicates, unclear state and federal expectations is another frequently cited 
impediment to successful local evaluation. This perception stems in large measure from the 
evolutionary legislative and regulatory history of the program. Since its enactment in 1 986, the 



'^Forty-five percent disagreed and 1 1 percent were neutral. 

?a 
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DFSCA has been amended no fewer than five times, resulting in significant changes to the scope 
of the program, including the addition of violence prevention as a key element. Not only has the 
legislation changed with some regularity from reporting period to reporting period, but the 
information requested by the federal government in its biennial surveys has changed with each 
survey administration. An imfortunate result of these, albeit necessary, modifications is that 
while LEAs and SEAs are aware of overall statutory requirements, they do not know until it is 
too late what specific types of evaluative information and data will be requested. 

Local fhistration over changing state and federal expectations for program evaluation is 
reflected in respondents' comments on how local evaluation could be improved, as are the other 
major impediments cited by program coordinators. Table 8 lists the most frequent 
recommendations offered by local DFSCA officials for improvement of local program 
evaluation. Each of these recommendations addresses to some extent one or more of the four 
greatest impediments to successful evaluation, as identified by local staff. 



Table 8: Estimated Percentage of Local DFSCA Coordinators With Recommendations for 
State or Federal Government Actions Needed to Improve Local Evaluation 



Recommendation 


Of Coordinators With 
Any Recommendations, 
Percentage Reporting 
Each Recommendation 
(n=l 1,835) 


Provide more training to district staff 


25% 


Provide more or better guidance on expectations 


21% 


Provide more funding 


.18% 


Provide more technical assistance or information generally 


15% 


Establish uniform data collection requirements 


14% 


Set aside a percentage of local funds for evaluation 


13% 


Provide a survey instrument for local use 


9% 


Streamline or reduce program/evaluation requirements 


8% 


Provide more flexibility in program/evaluation requirements 


5% 
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Perhaps the most critical of these recommendations, or the one action that would most 
positively influence the quality of local evaluations, is a clarification of federal expectations for 
program evaluation. Structuring the evaluation process such that whatever local effort is 
expended on evaluation is appropriately targeted to meet state and federal, as well as local, needs 
offers the best potential for improvement of local evaluations. Training and technical assistance 
that LEAs need to fully implement evaluation requirements could then also be focused to fit 
specific objectives and procedures. Moreover, the quality of local evaluation will improve to the 
extent federal expectations remain stable from one year to the next. 

ED's current effort to establish and implement national performance indicators for the 
program will go a long way toward clarifying federal expectations for local, as well as state- 
level, program performance and for the types of information required to evaluate that 
performance. While performance indicators are unlikely to address all local information needs, 
once model data collection standards are established and local program administrators know the 
criteria by which the success of their program will be measured, evaluation activity may be 
structured to collect these data in a routine and systematic fashion. Comparability of data across 
states and localities would also be enhanced if LEAs were encouraged to use a specific survey 
instrument for measuring the prevalence of AOD use. Model data collection standards and a 
recommended AOD use survey instrument would provide a framework upon which LEAs could 
include additional data collection tailored to local needs for performance information. 
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LOCAL USES OF DRUG-FREE SCHOOLS AND COMMUNITIES ACT 
PROGRAM EVALUATION MATERIALS AND 
TECHNICAL ASSISTANCE 



TELEPHONE INTERVIEW WITH 
LOCAL EDUCATIONAL AGENCY OFFICIALS 



Tabulated Responses 



TELEPHONE INTERVIEW WITH 
LOCAL EDUCATIONAL AGENCY OFFICIALS 



What is your job title? (n=l 3,762) 



a. 


Prevention program coordination 


24% 


b. 


District superintendent or assistant superintendent 


19% 


c. 


School counselor, psychologist, or social worker 


16% 


d. 


School principal or assistant principal 


13% 


e. 


Other local administrator 


10% 


f. 


Instructor 


7% 


g- 


Director of student services 


5% 


h. 


Student assistant program coordinator 


3% 


i. 


Other 


3% 


la. 


Does your district have a research and evaluation division? (n— 



01 


Yes 


6% 


02 


No 


93% 


03 


Do not know 


1% 



lb. (SKIP THIS QUESTION IF THE RESPONDENT'S JOB TITLE IS EVALUATION 
SPECIALIST.) Does your district employ or contract with an evaluation specialist? 

(n=l 3,273) 

01 Yes 13% 

02 No 87% 



A-1 




2. Now I'm going to ask you about some evaluation methods your district may have used to gather 
information about your prevention program during the last two years. For each method I 
mention, please indicate whether the evaluation activity has (1) been conducted to fulfill state 
requirements, (2) been conducted for local purposes only, or (3) not been used at all in the last 
two years. READ a-g AND CIRCLE ONE RESPONSE FOR EACH LINE. READ 
SECTIONS IN PARENTHESES UNDER b, c, AND g ONLY IF RESPONDENT SEEMS 
UNCLEAR ABOUT WHAT THE ITEM MEANS. NOTE: If information is used for both 
state and local purposes, circle ''01.'') 





State 


Local 


Not 


Don’t 




Mandated 


Use Only 


Collected 


Know 


Description of program activities (n= 12,749) 


76% 


17% 


3% 


4% 


Assessing how well the program is implemented 
(includes impressions of students or staff regarding 
the quality of programs or services; for example, 
evaluation of a training program, questionnaires 
collected from participants at the close of a special 
event regarding their reactions) (n=l 3,685) 


66% 


27% 


6% 


1% 


Longitudinal data collection of outcome measures 
(includes repeated measures on the same group of 
students; for example, administering student drug use 
surveys to the same group of students as they progress 
through various grades) (n= 13,676) 


31% 


22% 


46% 


1% 


Cross sectional data collection of outcome measures 
(n=13,616) 


27% 


23% 


48% 


2% 


Comparison of pre and post assessments of students 
(n=13,594) 


31% 


24% 


34% 


1% 


Comparison of outcome measures for local students with 
national or state averages (n= 13,680) 


27% 


27% 


44% 


2% 


Comparison of outcome measures for a treatment group 
(students receiving the program being evaluated) 
and a control/comparison group (students who do not 
receive the program being evaluated) (n=l 3,688) 


4% 


6% 


88% 


2% 



2G 
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Now I'd like to ask you about the types of data your district may have collected during the last 
two years: Again, for each type of data please indicate whether it has (1) been collected in 
response to state requirements, (2) been collected for local purposes only, or (3) not been 
collected during the last two years. (READ a-k AND CIRCLE ONE RESPONSE FOR 
EACH LINE. NOTE: If information is used for both state and local purposes, circle 

" 01 .") 







State 


Local 


Not 


Don’t 






Mandated 


Use Only 


Collected 


Know 


a. 


Local surveys of student use of alcohol and other 
(n=l3,689) 


48% 


38% 


13% 


1% 


b. 


Number of school disciplinary actions regarding 
alcohol and other drugs (n= 13,659) 


51% 


35% 


14% 


— 


c. 


Number of students referred for alcohol and other 
drug treatment (n= 13,666) 


41% 


40% 


19% 


1% 


d. 


Source of referral (n= 13,561) 


22% 


43% 


34% 


1% 


e. 


Number of juvenile arrests and convictions for violent 
crime or drugs (n=13,742) 


33% 


26% 


40% 


1% 


f. 


Extent of illegal gang activity (n= 13,728) 


20% 


21% 


59% 


1% 


g- 


Number of dropouts (n= 13,602) 


68% 


18% 


13% 


1% 


h. 


Rates of expulsions or suspensions from school 
(n= 13,676) 


61% 


28% 


10% 


1% 


i. 


Truancy/school absenteeism (n= 13,572) 


58% 


33% 


8% 


— 


j- 


Suicides and attempted suicides (n= 13,635) 


21% 


41% 


37% 


1% 


k. 


Number of students participating in activities for 
prevention and treatment of alcohol and other 
drug use (n= 13,6 12) 


53% 


31% 


16% 


1% 
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4. Has your district used evaluation data in any of the following ways? (READ a-e AND 
CIRCLE ONE RESPONSE FOR EACH LINE.) (n= 13,681) 







Yes 


Nq 


a. 


To justify continued funding 


84% 


16% 


b. 


To identify needs of students 


97% 


3% 


c. 


To direct changes in program content and delivery 


90% 


10% 


d. 


To fulfill state requirements 


93% 


7% 


e. 


To fulfill local requirements 


82% 


18% 


f 


Other 


2% 


98% 



5. Now I would like to ask about your perceptions of the role of evaluation, and about obstacles to 
successful evaluation. I will read a series of sentences and ask you to indicate whether you 
agree, disagree (or are neutral or undecided). (READ EACH SENTENCE AND CIRCLE 
ONE RESPONSE FOR EACH LINE) 



Agree Neutral Disagree 

a. My district is able to meet its evaluation needs 
through ongoing observation of program operations 
and activities; formal evaluation is not necessary. 

(n=13,748) 44% 11% 45% 

b. Our district would rather allocate all of its DFSCA 
resources to program development and student services 

than allocate up to five percent to evaluation. (n=13,762) 53% 11% 36% 

c. Our district prefers to have its DFSCA programs 
evaluated by an independent entity, 

rather than by program or district staff. (n=13,775) 14% 14% 72% 

d. Evaluation is necessary primarily as a means to 
document what we are doing for the state or the 
federal government (that is, for accountability) rather 

than for meeting local needs. (n=13,665) 26% 10% 64% 

e. It is realistic to expect DFSCA program staff 
to have the expertise needed for planning and 

conducting formal program evaluation. (n=13,755) 59% 12% 29% 

f. Formal evaluation is an essential component of a 

successful drug and alcohol prevention program. (n=13,762) 74% 10% 15% 
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6. Does your district need training or technical assistance in any of the following areas of program 
evaluation for your prevention program? (READ a - f AND CIRCLE ONE RESPONSE FOR 
EACH LINE.) (n=l 3,606) 







Yes 


No 


a. 


Evaluation planning 


61% 


39% 


b. 


Obtaining or developing data collection instruments 


60% 


40% 


c. 


Data collection 


40% 


60% 


d. 


Data analysis 


59% 


41% 


e. 


Reporting evaluation findings 


43% 


57% 


f 


Using evaluation findings for program improvement 


54% 


46% 



7. In which one of these areas that we just discussed are you most in need of training or assistance? 
(Read 01-06. CIRCLE ONLY ONE.) (n=l 0,203) 



01 Evaluation planning 


24% 


02 Obtaining or developing data collection instruments 


31% 


03 Data collection 


6% 


04 Data analysis 


12% 


05 Reporting evaluation findings 


4% 


06 Using evaluation findings for program improvement 


22% 


07 Other area 


1% 



8. Do any of the following factors represent significant impediments to successful evaluation of 
your district’s DFSCA program? (READ a-f AND CIRCLE ONE RESPONSE FOR EACH 
LINE.) (n= 13,762) 







Yes 


No 


a. 


Lack of money 


69% 


31% 


b. 


Lack of time 


81% 


19% 


c. 


Lack of expertise 


47% 


53% 


d. 


Confidentiality issues 


23% 


77% 


e. 


Unclear expectations from state/federal government 


45% 


54% 


f 


Lack of support from district officials/low priority 


24% 


76% 
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9. Which of the factors that we just discussed do you think is the greatest single impediment? 
(Read 01-06. CIRCLE ONLY ONE) (n=12,751) 



01 Lack of money 36% 

02 Lack of time 39% 

03 Lack of expertise 6% 

04 Confidentiality issues 4% 

05 Unclear expectations from state/federal government 10% 

06 Lack of support from district officials/low priority 6% 



10. Please indicate if during the last year your district obtained training or technical assistance in 
evaluation of its DFSCA programs from any of the following sources, and if so, briefly describe 
the type(s) of assistance or training provided. (***READ a-f.*** CIRCLE ONE 
RESPONSE ON EACH LINE AND RECORD TYPE OF ASSISTANCE RECEIVED 
FROM EACH SOURCE.) 





Xes 


N2 


College or university (specify type of assistance) (n=13,719) 
GEN’L WKSP=47%; DO EVAL.=I8%; DATA COLLECTION=15% 


14% 


86% 


State education agency (specify type of assistance) (n=13,707) 
GEN’L WKSP=66%; HELP WITH GRANTS=15%; DATA COLLECTION=7% 


50% 


50% 


Other state agency (specify type of assistance) (n= 13,665) 
GEN’L WKSP=47%; DARE=19%; DATA COLLECTION=8% 


20% 


80% 


DFSCA Regional Centers (specify type of assistance) (n= 13,648) 
GEN’L WKSP=43%; CURRICULUM MATER1ALS=13%; CONSULT=7% 


26% 


74% 


Independent professional (specify type of assistance) (n=13,485) 
GEN’L WKSP=28%; DO EVAL.=10%; STUDENT ASSISTANT=9% 


24% 


76% 


Other (specify type of assistance) (n=l 1 ,369) 
GEN’L WKSP=26%; DARE=1 8%; DO EVAL=8% 


24% 


76% 



1 1 Which of these sources that we just discussed provided the most useful assistance? (Read OI- 
OS. CIRCLE ONLY ONE) (n=l 0,25 5) 



01 College or university 


4% 




02 State education agency 


39% 




03 Other state agency 


9% 




04 Regional Centers for DFSCA 


12% 




05 Independent professional 


18% 




06 Other (n=l, 738) 


17% 




a. County education office 




20% 


b. Local law enforcement or other local agency 




14% 


c. None were helpful 




21% 


d. Other 




45% 



12. In Fall 1993 the U.S. Department of Education disseminated a handbook for DFSCA program 
evaluation to your district, entitled "Understanding Evaluation: The Way to Better Prevention 
Programs." Are you familiar with this handbook? (n=l 3,693) 

01 Yes 33% 

02 No 67% (If respondent says no, probe by describing the appearance and content 

of the book as follows: 

It has a purple and black cover, and was written by Lana Muraskin. 
It describes the steps in designing program evaluations and in 
implementing an evaluation design, including an example using a 
fictitious school district called Wood County. Does that help? 

If respondent is still not aware of the handbook skip to item 21.) 



13. Have you read the handbook? (n=4,55 1) 

01 Yes 65% 

02 No (Skip to item 22) 35% 



14. Did you receive training in how to use the evaluation handbook? (n=2,910) 



16% 

84% 
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01 Yes 

02 No 



15. Who provided the training? (n=492) 

SEA=50%; REGIONAL CENTERS=26%; UNIV.=1 1%; OTHER=l% 

16. What do you like most about the handbook? (n=2,934) 



a. 


Easy to understand 


17% 


b. 


Practical orientation 


16% 


c. 


Use of case examples 


8% 


d. 


Provides a good overview 


18% 


e. 


Resource/reference 


6% 


f 


Nothing specific 


9% 


g- 


Do not remember 


27% 



1 7. What do you like least about the handbook? (n-2,934) 



a. 


Not practical enough 


4% 


b. 


Too long 


3% 


c. 


Examples do not apply to my program 


5% 


c. 


Nothing specific 


18% 


d. 


Do not remember 


9% 


f 


Other 


58% 



1 8. Have you or your colleagues ever used the handbook or applied information from the handbook 
in your work? (n=2,753) 

01 Yes 64% 

02 No (Skip to item 22) 34% 

03 Don’t know 2% 



1 9. In which of the following ways have you used the handbook? (READ a-g AND CIRCLE ONE 



RESPONSE ON EACH LINE) (n= 1,758) 







Yes 


Nfi 


a. 


In overall evaluation planning or design 


88% 


12% 


b. 


To help identify evaluation questions or issues 


79% 


21% 


c. 


In developing data collection instruments 


45% 


55% 


d. 


To help guide implementation of the evaluation design 


58% 


42% 


e. 


As an aid to interpreting data collected 


62% 


38% 


f 


To help guide reporting of evaluation findings 


59% 


41% 


g- 


As part of staff training 


47% 


53% 


h. 


Other use 


60% 


40% 
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20. Overall, how useful has the handbook been to you and district staff in evaluating your DFSCA 
programs? (n= 1,803) 



01 


Very useful 


22% 


02 


Somewhat useful 


76% 


03 


Not useful 


1% 


04 


Don’t know 


1% 



21 . Please briefly explain your response: (n=l,658) 



a. 


Good reference 


15% 


g. Information was applicable 


3% 


b. 


Help in planning 


12% 


h. Information not applicable 


3% 


c. 


No specific reason 


13% 


i. Confirmed our approach 


3% 


d. 


Need training also 


3% 


j. Used with other resources 


5% 


e. 


Too technical 


3% 


k. Other 


19% 


f 


Examples 


4% 







22. Are you familiar with any of these other technical assistance products? (READ a-e AND 
CIRCLE ONE RESPONSE ON EACH LINE) (n=l 3,748) 



Yes No 

a. Handbook for Evaluating Drug and Alcohol Prevention 

Programs (STEPP) (University of Maryland, 1 990) 1 6% 84% 

b. Program Evaluation Handbook: Drug Abuse Education, 

lOX Associates 93% 

c. Prevention Plus III, Assessing Alcohol and Other 

Drug Prevention Programs at the School and Community 

Level (Linney and Wandersman) 20% 80% 

d. Evaluation of Prevention Programs: A Basic Guide 

for Practitioners (M.ohQ\%, \9%A) 12% 88% 

e. CSAP Technical Report #8: Measurements and Prevention 7% 93% 

f. Other 2% 98% 



23. Of the products we just discussed, which one is the most helpful with respect to planning and 
conducting program evaluations? (CIRCLE ONLY ONE) (N=4,986) 



01 Handbook for Evaluating Drug and Alcohol Prevention Programs (STEPP) 1 7% 

02 Program Evaluation Handbook: Drug Abuse Education 10% 

03 Prevention Plus III, Assessing Alcohol and Other Drug Prevention 

Programs at the School and Community Level 26% 

04 Evaluation of Prevention Programs: A Basic Guide for Practitioners 11% 

05 CSAP Technical Report #8 - Measurements and Prevention 5% 

06 Other 30% 



24. Briefly explain why this resource is helpful. (n=4,986) 



a. 


Easy to understand 


5% 


b. 


Provides useful tools and instruments 


8% 


c. 


Practical orientation 


1% 


d. 


Use of case examples 


8% 


e. 


Good reference 


9% 


f 


Applicable to local program 


3% 


g- 


Most readily available resource 


3% 


h. 


Helpful in project design 


3% 


i. 


Provides models, step-by-step guidance 


6% 


j- 


No specific reason 


29% 


k. 


Other 


20% 


1. 


Don’t know 


18% 



25. What can your state education agency or the U.S. Department of Education do to improve local 
evaluations of DFSCA programs? (n=13,761) 

a. Provide more or better guidance on expectations 1 8% 

b. Establish uniform data collection requirements 12% 

c. Set aside a percentage of local funds for evaluation 1 1 % 

d. Provide training to district staff 22% 

e. Provide more flexibility in program/evaluation requirements 5% 

f. Streamline-reduce program/evaluation requirements 7% 

g. Strengthen evaluation requirements 3% 

h. Provide more funding 1 5% 

i. Disseminate effective evaluation models/programs 2% 

j. Provide more guidance on parent and community involvement 3% 

k. Help conduct evaluations/do evaluations for us 3% 

l. Provide more technical assistance or information generally 13% 

m. Provide more guidance on parental consent/confidentiality 1% 

n. Provide a survey instrument for us to use 8% 

o. More time 4% 

p. None/don’t know 14% 

q. Other 10% 
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